Restoring distributed shared memory data consistency within a recovery process from a cluster node failure

ABSTRACT

A DSM component is organized as a matrix of page. The data structure of a set of data structures occupies a column in the matrix of pages. A recovery file is maintained in a persistent storage. The recovery file consists of entries and each one of the entries corresponds to a column in the matrix of pages by a location of each one of the entries.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a Continuation of U.S. patent application Ser. No.13/801,797, filed on Mar. 13, 2013, which is a Continuation of U.S.patent application Ser. No. 12/950,869, filed on Nov. 19, 2010.

FIELD OF THE INVENTION

The present invention relates in general to computers, and moreparticularly to apparatus, method and computer program productembodiments for restoring distributed shared memory (DSM) dataconsistency within a recovery process from a failure of a node in acluster of nodes in a computing environment.

DESCRIPTION OF THE RELATED ART

Distributed shared memory (DSM) provides an abstraction that allowsusers to view a physically distributed memory of a distributed system asa virtually shared address space. A central advantage of DSM is theconvenience provided for the programmers of distributed applications,reducing or eliminating the requirement to be aware of the distributedarchitecture of the system and the requirement to use a less intuitiveform of communication within a distributed system via message passing.Another advantage of DSM is the ability to directly port softwarewritten for non-distributed systems to work on distributed systems.

There are many forms of DSM algorithms, all of them sharing afundamental architecture of being composed of distributed agentsdeployed on the clustered nodes (i.e. processors), each agentmaintaining local data structures and memory data segments, and acommunication protocol over a message passing layer being used tocoordinate operations among the agents. A central goal of any DSMalgorithm is to minimize message traffic for a given workload. Anunderlying goal is to maintain memory coherency of each agent and of theentire DSM.

SUMMARY OF THE INVENTION

In an application based on DSM, where data entities residing on DSMmemory pages may be accessed intensively and interchangeably for writingand reading by multiple users across a DSM cluster, the DSM algorithmsof the class described above, although optimized in themselves, mayentail an intense messaging traffic, as pages are frequently invalidatedto provide exclusive access and then shared again among respectivecluster nodes.

Furthermore, such a usage pattern may make the process of failover morechallenging and more costly in terms of the required processing time andmessaging traffic. Failover is the process where one or more of thenodes in the cluster fail and eventually removed from the cluster, andthe remaining nodes recover their internal and mutual consistency, suchthat normal operation is enabled after completion of the failoverprocess. One challenge that may arise with the usage pattern describedabove is that the latest contents of the set of DSM pages for which thefailed node was the last node to update their contents, may be lost andshould be recovered using some application specific method or logic. Inview of the foregoing, a need exists for mechanisms facilitatingreduction of DSM messaging traffic, and facilitating the process ofrecovery performed by remaining nodes during a failover operation.

Accordingly, various embodiments for facilitating search, list andretrieval operations on a persistent data set using a distributed sharedmemory (DSM) by a processor device are provided. In one embodiment,various aspects of the present invention are operable in the followingapplication. In this application, a set of data structures is stored onpersistent storage, which is shared among and accessible to a cluster ofprocessors (i.e. nodes). It is assumed that the data set fits in thememory of each of the cluster nodes, but the data set is required to bepersistent. There is an order relation among these data structures, andeach data structure is associated with a set of properties constitutinga key for identification and search. The key is constant once the datastructure has been created. Most frequent operations on the data setinclude search, list and retrieval of data structures. Less frequentoperations include update of the contents/properties of individual datastructures, and least frequent operations include addition and removalof data structures to/from the set. Each of these operations is appliedconcurrently to the data set.

In such an exemplary application, the mechanisms of the presentinvention aim, in one embodiment, to optimize and accelerate theprocessing of the frequent operations, primarily the search, list andretrieval operations. In these mechanisms, for example, a DSM is used tostore the data structures in a cluster shared memory, such that updates,deletions and insertions (which are infrequent) are applied to both thepersistent storage and the DSM, and search, list and retrievaloperations (which are frequent) are processed directly from the DSM,thus saving accesses to persistent storage and accelerating processing.

In an exemplary embodiment, a data structure in a persistent data set isstored in two separate DSM pages: a first page includes an identifyingsearch key of the data structure, the search key also pointing to alocation of the data structure in the storage. A second page includesthe updateable properties of the data structure. A DSM storing the datastructures' pages is organized as a two row matrix, wherein one rowstores the data structures' keys pages, and the second row stores thedata structures' updateable properties pages, and each occupied columnin the matrix corresponds to a specific data structure in the persistentstorage. A synchronization page is defined for the DSM matrix,facilitating synchronization between operations that modify theassignment of columns to data structures, and operations that do notmodify the assignment. Insertion operations take an exclusive permissionon the synchronization page for the duration of the operation, while allother types of operations take a shared permission on thesynchronization page for the duration of the operation. To facilitatefailover processing, a DSM recovery file is maintained in the persistentstorage, where the file reflects the structure of the DSM matrix. Thefile is updated during insertion operations, and is used during failoverprocessing to locate the persistent data structures to be loaded intoinvalid DSM pages.

In an exemplary embodiment, a DSM component is organized as a matrix ofpage. The data structure of a set of data structures occupies a columnin the matrix of pages. A recovery file is maintained in a persistentstorage. The recovery file consists of entries and each one of theentries corresponds to a column in the matrix of pages by a location ofeach one of the entries.

BRIEF DESCRIPTION OF THE DRAWINGS

In order that the advantages of the invention will be readilyunderstood, a more particular description of the invention brieflydescribed above will be rendered by reference to specific embodimentsthat are illustrated in the appended drawings. Understanding that thesedrawings depict embodiments of the invention and are not therefore to beconsidered to be limiting of its scope, the invention will be describedand explained with additional specificity and detail through the use ofthe accompanying drawings, in which:

FIG. 1 illustrates an exemplary distributed shared memory (DSM)architecture in which aspects of the present invention may beimplemented;

FIG. 2 illustrates exemplary architecture for organizing data structuresin a distributed shared memory matrix, in which various aspects of thepresent invention may be implemented;

FIG. 3 illustrates an exemplary method for performing a search procedurefor a data structure according to various aspects of the presentinvention;

FIG. 4 illustrates an exemplary method for performing a modificationprocedure of a data structure according to various aspects of thepresent invention

FIG. 5 illustrates an exemplary method for performing a deletionprocedure of a data structure according to various aspects of thepresent invention;

FIG. 6 illustrates an exemplary method for performing an insertionprocedure of a data structure according to various aspects of thepresent invention;

FIGS. 7A and 7B illustrate exemplary methods for performing a failoverprocedure according to various aspects of the present invention; and

FIG. 8 illustrates exemplary hardware for implementing one or moreaspects of the present invention.

DETAILED DESCRIPTION OF THE DRAWINGS

As previously mentioned, distributed shared memory (DSM) provides anabstraction that allows users to view a physically distributed memory ofa distributed system as a virtually shared address space. The entirememory space of a DSM instance is partitioned into pages of arbitraryand possibly different sizes. A DSM agent resides on each of theclustered nodes. Each DSM agent manages a local set of memory pages,corresponding to the cluster-wide DSM memory space, and an associatedtable of entries. Each page is associated with an entry in that table.

A user may obtain permission to access a specific page, via the DSMagent servicing its node, atomically with respect to all other users onall nodes sharing the DSM instance. The permission may be shared, namelythe data segment may be only read. This permission can be obtainedconcurrently by multiple users. Alternatively the permission may beexclusive, namely the data segment may be also modified. This permissionis mutually exclusive with all other users. A valid permission generallyrefers to having either shared or exclusive permission. When grantingpermission, the latest contents of the page is also made available tothe user (the local agent may obtain it from another agent over thenetwork). The latest permission obtained for each page is recorded byeach agent within its table of entries. Permissions are modified onlydue to user requests.

Each page has a single owner in the DSM instance, which is one of theDSM agents. The owner's identify (or approximate identify) for each pageis recorded by each agent within the local entry associated with thatpage. The ownership of a page may be dynamically changed duringprocessing of user requests, and different entries may have differentowners. Generally, when a user request entails an upgrade of permissionfor a specific page, a request must be issued to the owner of that page.If the owner is not the local agent then the request is issued over themessage passing layer to the remote owning agent. The owner of a pageserializes processing of requests issued concurrently for that page, andhas complete knowledge of the whereabouts of the latest contents of thatpage. Ownership of a page may be exchanged between the agents, triggeredby processing of user requests, in the following cases: (1) when a useris given exclusive permission on a page, the DSM agent servicing itsnode is set as the owner of that page, and (2) when a user is givenshared permission on a page and the remote owner does not have anypermission on that page, the DSM agent servicing the node of therequesting user is set as the owner of that page.

Generally, when a user requests permission on a page, which entailsupgrading the current permission held by the local DSM agent on therelevant page (upgrading means switching from no permission to shared orexclusive permission, or switching from shared permission to exclusivepermission), a message is generated and sent by the DSM agent servicingthe cluster node where the requesting user is operating. There aregenerally two types of messages: (1) a permission request: sent from anon owning agent to the agent owning the page, in order for the owningagent to process and grant the upgraded permission on that page, and (2)an invalidation request: sent from an agent owning a page to anotheragent, in a case where the owning agent requires to upgrade itspermission from shared to exclusive, and the other agent may hold ashared permission and a valid copy of that page. There may be multiplenon owning agents complying with this condition, and in such a case aninvalidation message is sent to each such agent.

A DSM algorithm defines procedures to be performed by the DSM agentsupon receiving an event. There are procedures for processing requestsissued by local users. These procedures may generate and send requestmessages to remote agents, in accordance with the cases in the previousparagraph. There are also procedures for processing requests encoded inmessages arriving from remote agents over the message passing layer.These procedures may generate and send response messages to the remoteagents.

All of these procedures include a first step that checks for conditionsthat require the procedure to block until these conditions are cleared.These conditions apply for the specific entry and page for which arequest is being processed by the procedure. One condition for blockingis mutual exclusion, namely enforcing a policy of multiple readers and asingle writer. An additional condition is blocking as long as there isan ongoing messaging session (i.e. a request followed by a response)initiated by the local agent with regards to the specific entry and pagewhich are being processed. After this first step, a procedure proceedsto process the request, which may or may not include generating andsending a message to one or more remote agents.

FIG. 1 illustrates exemplary DSM architecture 10 in which aspects of thepresent invention may be implemented. Two exemplary cluster nodes, A andN (denoted by reference numbers 12, and 14) are represented, in whichmessages 40 are passed between. Turning first to cluster node 12, anumber of applications 16, 18 are associated with a number of users(e.g., representative of 1 to n users) as shown. The number of users arerepresentative of threads of execution within the applications 16, 18,where the threads of execution perform logic that uses a DSM instance. ADSM agent 24 servicing the cluster node 12 is responsible for a table ofentries 28 and a set of pages 30. For the DSM agent 24, a number ofprocedures for handling local and remote requests 36 and 38 areassociated, where the number of procedures operate on the table ofentries 28 and the set of pages 30. Similarly, in cluster node 14, anumber of applications 20 and 22 are associated with a number of users(again, representative of 1 to n users). DSM agent 26 servicing thecluster node 14 is responsible for a table of entries 32 and a set ofpages 34. As similarly described for DSM agent 24, for the DSM agent 26,a number of procedures for handling local and remote requests 42 and 44are associated, where the number of procedures operate on the table ofentries 32 and the set of pages 34. A specific example of a DSMalgorithm of the class described above is specified in U.S. PatentApplication Publication 2010/0049922, entitled “Distributed SharedMemory”, and assigned to International Business Machines Corporation(IBM), the relevant portions therein incorporated by reference.

Again, and as previously described, in an application based on DSM,where data entities residing on DSM pages may be accessed intensivelyand interchangeably for writing and reading by multiple users across thecluster, the DSM algorithms of the class described above, althoughoptimized in themselves, may entail an intense messaging traffic, aspages are frequently invalidated to provide exclusive access and thenshared again among the nodes. Furthermore, such a usage pattern may makethe process of failover more challenging and more costly in terms of therequired processing time and messaging traffic.

In one embodiment, various aspects of the present invention are operablein the following application. In this application, a set of datastructures is stored on persistent storage, which is shared among andaccessible to a cluster of processors (i.e. nodes). It is assumed thatthe data set fits in the memory of each of the cluster nodes, but thedata set has to be persistent. There is an order relation among thesedata structures, and each data structure is associated with a set ofproperties constituting a key for identification and search. The key isconstant once the data structure has been created. Most frequentoperations on the data set include search, list and retrieval of datastructures. Less frequent operations include update of thecontents/properties of individual data structures, and least frequentoperations include addition and removal of data structures to/from theset. Each of these operations is applied concurrently to the data set.

In such an exemplary application, the mechanisms of the presentinvention aim, in one embodiment, to optimize and accelerate theprocessing of the frequent operations, primarily the search, list andretrieval operations. In these mechanisms, for example, a DSM is used tostore the data structures in a cluster shared memory, such that updates,deletions and insertions (which are infrequent) are applied to both thepersistent storage and the DSM, and search, list and retrievaloperations (which are frequent) are processed directly from the DSM,thus saving accesses to persistent storage and accelerating processing.

In addition to the foregoing, additional mechanisms of the presentinvention enable to achieve the following properties. First, efficientconcurrency of operations applied to the data set; second, minimizedinternal messaging traffic between the distributed agents of the DSMmechanism; and finally, efficient failover processing.

The basic DSM algorithm, whose main elements are described previously,optimizes and minimizes the internal communication between the DSMagents residing on the cluster nodes, based on a general usage patternof a DSM application. The illustrated embodiments enable to furtherreduce the internal communication between the DSM agents, by means ofarchitecting the DSM based application and the data structures itmaintains over DSM pages in a specific way. The mechanisms of theillustrated embodiments support efficient concurrency of operationsapplied to the data set, namely search, list, retrieval, update,insertion and deletion operations, while minimizing internal DSMcommunication as will be further detailed. Furthermore, the mechanismsof the illustrated embodiments support efficient failover processing.

In one aspect of the illustrated embodiments, each data structure in thepersistent set is stored in two separate pages in the DSM. One pagecontains the properties constituting the identification and search keyof the data structure, and the second page contains the updatableproperties of the data structure. The key also points to the location ofthe data structure in the storage. The DSM is organized basically as atwo-row matrix. One row stores the data structures' keys, and the secondrow stores the data structures' updateable properties, such that eachoccupied column (i.e. one key page and one updateable properties page)in the matrix corresponds to a specific data structure in the persistentset. The data structures are stored in the DSM matrix in a sorted orderbased on their keys. This allows to locate specific data structuresefficiently using binary search, and to efficiently process listoperations, which are normally based also on the keys order. The size ofthe DSM matrix is defined such that it enables to accommodate themaximal number of data structures supported in the persistent set.Assignment of a two-page column in the DSM matrix to a specific datastructure may change over time, depending on the operations applied tothe data set (e.g. insertion and deletion of data structures).

In another aspect of the illustrated embodiments, a synchronization pageis defined for the DSM matrix, facilitating synchronization betweenoperations that modify the assignment of columns to data structures, andoperations that do not modify the assignment. Essentially, the only typeof operations that modify this assignment is insertion of datastructures. Thus insertion operations take an exclusive permission onthe synchronization page for the duration of the operation, while allother types of operations take a shared permission on thesynchronization page for the duration of the operation. Thesynchronization page may contain additional information (depending onthe specific implementation), such as a version number of the DSMmatrix.

As previously mentioned, the mechanisms of the present invention supportefficient failover processing. A challenge in processing failover inthis setting is the case where a node in the cluster performs changes inthe assignment of DSM columns to persistent data structures, and thenfails before other nodes access these pages, thus the contents of thesepages and their assignment information are lost in the perspective ofthe other nodes during failover. Specifically, in this case, at the timeof failover processing, a subset of the key pages are invalidated in thelocal DSMs of all remaining cluster nodes (as these pages were lastupdated by the node that failed), and thus there is no information onwhich data structures should be loaded into these pages, and where tolocate these structures in the storage. To solve this challenge, themechanisms of the illustrated embodiments define a DSM recovery file inthe persistent storage, which reflects the structure of the DSM matrix.Specifically, each entry in the file corresponds by its location to aspecific column in the DSM matrix, and indicates whether or not theassociated column in the DSM matrix is assigned with a persistent datastructure, and if assigned it indicates the specific location of thatdata structure in the persistent storage. The recovery file is updatedduring insertion operations (which change columns' assignments), and isused during failover processing to locate the persistent data structuresto be loaded into invalid DSM pages.

The benefits of the illustrated embodiments, following, includeincreased efficiency of operations that are based solely on the keyportions of the DSM based data structures, namely search, list andretrieval operations, whose processing essentially requires no internalDSM communication. In addition, the DSM matrix exemplified in theillustrated embodiments efficiently supports concurrent operations,namely search, list, retrieval, update of data structures, insertion anddeletion of data structures, such that internal DSM communication isminimized. And finally, as previously mentioned, the illustratedembodiments efficiently implement failover processing.

Turning now to FIG. 2, exemplary architecture 50 illustrating a DSMcontaining a number of data structures (A to N) in columns is shown. Thearchitecture 50, and the various data structures are organized in theDSM by key pages and by updatable properties pages in a two-row matrix52 as shown, and as described previously. A synchronization page,defined in the DSM and associated with the DSM matrix, assists infacilitating synchronization between operations that modify theassignment of columns to data structures, and operations that do notmodify the assignment as previously described. The DSM architecture 50is operable in conjunction with persistent storage 54, which houses thedata structures A-N as shown, and also includes a recovery file 56having 1 to N entries showing the locations of the data structures A-Nin the persistent storage 54.

FIGS. 3-7, following, illustrate exemplary methods for processingvarious operations such as a search, modification, insertion, deletionand failover operation as previously described. As one skilled in theart will appreciate, various steps in the methods shown in the followingfigures may be implemented in differing ways to suit a particularapplication. In addition, the described method may be implemented byvarious means, such as hardware, software, firmware, or a combinationthereof operational on or otherwise associated with the computingenvironment. For example, the methods presented, following, may beimplemented, partially or wholly, as a computer program productincluding a computer-readable storage medium having computer-readableprogram code portions stored therein. The computer-readable storagemedium may include disk drives, flash memory, digital versatile disks(DVDs), compact disks (CDs), and other types of storage mediums.

Turning first to FIG. 3, an exemplary method 60 for performing a searchprocedure for a data structure in an architecture such as architecture50 (FIG. 2) is depicted. Method 60 begins (step 62), by a sharedpermission taken on the synchronization page for the duration of theoperation (step 64). A binary search is performed using sharedpermissions to key pages (step 66). Page permissions are not nested. Asa next step, at the end of the search operation, the shared permissionson the key pages, and then on the synchronization page, are released(step 68). The method 60 then ends (step 70).

FIG. 4 shows an exemplary method 80 for performing a modificationprocedure of a data structure in an architecture such as architecture 50(again, FIG. 2). The modification is applied to the updatable portion ofthe data structure, and the operation does not modify the structure ofthe DSM matrix. Method 80 begins (step 82) with a shared permissiontaken on the synchronization page for the duration of the operation(step 84). If the following processing of the operation depends in anyway on the knowledge whether a structural modification has occurredsince a last access to the DSM matrix (for example if processing relieson some local information which depends on the structure of the DSMmatrix), then the version number in the synchronization page is comparedto a local version number, and if the local version number is smallerthan that in the synchronization page (step 86), then logic is activatedto refresh the local knowledge (such logic is generally based onread-only accesses to some subset of the DSM matrix) (step 88). Thelocal version number is then set to be the version number in thesynchronization page.

The requested data structure is located using a binary search on the keypages, or alternatively relying on other logic based on informationexternal to that stored in the DSM (step 90). This is done using sharedpermissions on the DSM key pages. An exclusive permission is taken onthe page containing the updatable portion of the data structure to bemodified (step 92). Page permissions are not nested. The page's contentis modified (step 94). The associated persistent data structure ismodified (step 96). The exclusive permission on the modified page isreleased (step 98). The shared permission on the synchronization page isreleased (step 100). The method 80 then ends (step 102).

FIG. 5, following, shows an exemplary method 110 for performing adeletion procedure of a data structure in an architecture such asarchitecture 50 (again, FIG. 2). Method 110 begins (step 112), by takinga shared permission on the synchronization page for the duration of theoperation. If the following processing of the operation depends in anyway on the knowledge whether a structural modification has occurredsince a last access to the DSM matrix (for example if processing relieson some local information which depends on the structure of the DSMmatrix), then the version number in the synchronization page is comparedto a local version number, and if the local version number is smallerthan that in the synchronization page (step 116), then logic isactivated to refresh the local knowledge (such logic is generally basedon read-only accesses to some subset of the DSM matrix) (step 118). Thelocal version number is then set to be the version number in thesynchronization page.

The data structure to be deleted is located using a binary search on thekey pages, or alternatively relying on other logic based on informationexternal to that stored in the DSM. This is done using sharedpermissions on the DSM key pages (step 120). An exclusive permission istaken on the page containing the key portion of the data structure to bedeleted (step 122). Page permissions are not nested. The data structureis marked in its key page as not assigned (step 124). The data structureis removed from its location in the persistent storage (step 126). Theexclusive permission on the key page is released (step 128). The sharedpermission on the synchronization page is released (step 130). Themethod 110 then ends (step 132).

It should be noted that since deletion is implemented by marking thedata structure as deleted, and avoiding any structural modification ofthe DSM matrix (namely, other pages' assignments with data structuresare not modified by this operation), the deletion can take sharedpermission on the synchronization page. All other operations, whenaccessing key pages, check if the key page is assigned to a datastructure or not, and consider the column only if the key page isassigned. Also, it should be noted that the DSM recovery file is notaccessed during the deletion operation. After the deletion operation theDSM recovery file still contains a reference to the data structure thathas been deleted. However, since the data structure is deleted from thepersistent storage this state is consistent, as will be shown in theexemplary method for processing failover, following.

FIG. 6, following, shows an exemplary method 140 for performing aninsertion procedure of a data structure in an architecture such asarchitecture 50 (again, FIG. 2). Method 140 begins (step 142), by takingan exclusive permission on the synchronization page for the duration ofthe operation (step 144). If the following processing of the operationdepends in any way on the knowledge whether a structural modificationhas occurred since a last access to the DSM matrix (for example ifprocessing relies on some local information which depends on thestructure of the DSM matrix), then the version number in thesynchronization page is compared to a local version number, and if thelocal version number is smaller than that in the synchronization page(step 146), then logic is activated to refresh the local knowledge (suchlogic is generally based on read-only accesses to some subset of the DSMmatrix) (step 148). The version number in the synchronization page isthen incremented, and this incremented value is also set to be the localversion number.

An appropriate location for the data structure to be inserted is locatedusing a binary search on the key pages, or alternatively relying onother logic based on information external to that stored in the DSM(step 150). This is done using shared permissions on the DSM key pages.Data structures are moved to create an empty column for the new datastructure, generally until reaching a column marked as deleted (in itskey page) and overwriting it (step 152). This is done using exclusivepermissions on key portion pages and updatable portion pages. The newdata structure is inserted, using exclusive permissions on the key andupdatable portion pages of the empty column assigned to the new datastructure (step 154). Throughout this process page permissions are notnested.

The entry in the DSM recovery file, which is associated with the columnin the DSM matrix where the new data structure has been inserted, isupdated with the key properties of the new data structure and a pointerto the determined location of that data structure in the persistentstorage. In addition, for all other columns in the DSM matrix, whoseassignments with persistent data structures have been modified, theirrespective entries in the DSM recovery file are also modified with thekey values and storage pointers reflecting the new assignments.Modification of the recovery file is done such that the updates areperformed in memory and then these updates are flushed to the persistentfile (step 156). The new data structure is now written to the determinedlocation in the persistent storage (step 158). The exclusive permissionson the key and updatable portion pages of the column assigned to the newdata structure are released (step 160). The exclusive permission on thesynchronization page is released (step 162). The method 140 then ends(step 164).

As previously mentioned, a failover event occurs when a node in thecluster fails. In failover processing, the DSM and the persistent dataare recovered to reach consistency, and when this processing is completethe cluster is operational. Turning now to FIGS. 7A and 7B, an exemplarymethod 170 for processing a failover operation in architectures such asarchitecture 50 (again, FIG. 2) is shown. Method 170 begins, as apreliminary matter, by the selection of a node (step 174). One node inthe cluster performs the recovery process. This node is eitherpredetermined or selected at the beginning of failover processing.

In the recovery process, each page in the DSM is inspected, and a pagespecific recovery operation is required only for pages whose content isinvalid in the context of recovery. An invalid page in the context ofrecovery is defined as a page that is invalid in the local DSMs of allthe remaining cluster nodes (i.e. all the nodes except the node thatfailed). To compute the invalid pages (step 176), for each page which isinvalid in the local DSM of the node performing the recovery, this nodeinquires with the other nodes regarding the validity of that page, andthe page is declared as invalid in the context of recovery only if it isinvalid in the DSMs of all the remaining cluster nodes. This process maybe performed with consolidated messages (i.e. containing information onmultiple invalid pages). For pages whose content is invalid in thecontext of recovery, in this exemplary embodiment, the DSM mechanismblocks all accesses to such pages for the duration of failoverprocessing, for all regular threads of execution (i.e. threads which arenot the thread performing the recovery). Therefore, the threadperforming the recovery can access these pages, after ensuring thattheir content is invalid in the context of recovery, without obtainingpermission for their access.

The failover procedure iterates on the columns of the DSM matrix (e.g.,step 178), and for each column (step 200) it first inspects the key page(step 180) and then the updateable portion page (step 190). If thecurrent page is a key page (again, step 180), and the page is invalid(step 182), then its associated entry in the recovery file is inspectedand the following is applied. If the entry indicates that there is noassignment of this column in the DSM matrix to a data structure in thepersistent storage, or if the entry does indicate that there is such anassignment but there is no actual data structure in the pointed locationin the persistent storage (step 184), then that key page is reset to beunassigned, and the key page is also set as valid (steps 188). It isnoted that since the recovery file is not updated during deletion ofdata structures, there may be cases where an entry in the recovery fileindicates assignment to a persistent data structure, but that datastructure does not exist in the pointed location in the persistentstorage.

If the entry indicates that there is an assignment of this column in theDSM matrix to a data structure in the persistent storage, and there isan actual data structure in the pointed location in the persistentstorage (again, step 184), then the content of that data structure isloaded from the persistent storage into the key page, and the key pageis set as assigned to that data structure. The key page is set as valid(step 186).

If the current page is an updateable portion page (step 190), and thepage is invalid (step 192), then the following is applied. If the keypage associated with the respective column of that updateable portionpage is unassigned to a data structure (step 194), then the updateableportion page is also reset to be unassigned, and the updateable portionpage is set as valid (step 198). If the key page associated with therespective column of that updateable portion page is assigned to a datastructure (again, step 194), then the contents of that data structure isloaded from the persistent storage into the updateable portion page, andthe updateable portion page is set as assigned to that data structure.The updateable portion page is set as valid (step 196).

The failover procedure inspects the synchronization page (step 202), andif it is invalid then the following is applied. If the synchronizationpage contains information then this information is restored using anapplication specific method. For example, if the synchronization pagecontains a version number, then this version number is set with a valuethat is higher than the locally stored version value. Thesynchronization page is set as valid (step 204). The method 170 thenends (step 206).

Turning now to FIG. 8, an exemplary portion 212 of a DSM system 210 isillustrated. Portion 212 of DSM system 210 is operable in a computerenvironment as a portion thereof, in which mechanisms of the followingillustrated embodiments may be implemented. It should be appreciated,however, that FIG. 8 is only exemplary and is not intended to state orimply any limitation as to the particular architectures in which theexemplary aspects of the various embodiments may be implemented. Manymodifications to the architecture depicted in FIG. 8 may be made withoutdeparting from the scope and spirit of the following description andclaimed subject matter.

Portion 212 includes a processor 214 and a memory 216, such as randomaccess memory (RAM). The DSM system 210, and portion 212 may beoperatively coupled to several components not illustrated for purposesof convenience, including a display, which presents images such aswindows to the user on a graphical user interface, a keyboard, mouse,printer, and the like. Of course, those skilled in the art willrecognize that any combination of the above components, or any number ofdifferent components, peripherals, and other devices, may be used withthe DSM system 210 and/or portion 212.

In the illustrated embodiment, the portion 212 operates under control ofan operating system (OS) 218 (e.g. z/OS, OS/2, LINUX, UNIX, WINDOWS, MACOS) stored in the memory 216, and interfaces with the user to acceptinputs and commands and to present results. In one embodiment of thepresent invention, the OS 218 facilitates DSM functionality according tothe present invention. To this end, OS 218 includes a DSM module 228which may be adapted for carrying out various processes and mechanismsin the exemplary methods described following.

Portion 212 may implement a compiler 222 that allows an applicationprogram 220 written in a programming language such as COBOL, PL/1, C,C++, JAVA, ADA, BASIC, VISUAL BASIC or any other programming language tobe translated into code that is readable by the processor 214. Aftercompletion, the application program 220 accesses and manipulates datastored in the memory 216 of the system 210 using the relationships andlogic that was generated using the compiler 222.

To further implement and execute mechanisms and processes according tothe present invention, OS 218, in conjunction with memory 216, processor214, application program 220, and other computer processing, networking,and storage components, may implement DSM agents 224 and DSM algorithms226 employing various aspects of the previously described exemplaryembodiments. As the skilled artisan will appreciate, the mechanisms ofDSM algorithms 226 and DSM agents 224 as presently illustrated may beimplemented in various forms and architectures. Accordingly, theillustration of DSM algorithms 226 and DSM agents 224 in the presentfigure is again intended to demonstrate logical relationships betweenpossible computing components in the DSM system 210 and portion 212, andnot to imply a specific physical structure or relationship.

In one embodiment, instructions implementing the operating system 218,the application program 220, and the compiler 222, as well as the DSMalgorithms 226 and DSM agents 224 are tangibly embodied in acomputer-readable medium, which may include one or more fixed orremovable data storage devices, such as a zip drive, disk, hard drive,DVD/CD-ROM, digital tape, solid state drives (SSDs), etc. Further, theoperating system 218 and the application program 220 may compriseinstructions which, when read and executed by the system 210 and/orportion 212, cause the system 210 and/or portion 212 to perform thesteps necessary to implement and/or use the present invention.Application program 220 and/or operating system 218 instructions mayalso be tangibly embodied in the memory 216. As such, the terms “articleof manufacture,” “program storage device” and “computer program product”as may be used herein are intended to encompass a computer programaccessible and/or operable from any computer readable device or media.

Embodiments of the present invention may include one or more associatedsoftware application programs 220 that include, for example, functionsfor managing a distributed computer system comprising a network ofcomputing devices, such as a storage area network (SAN). Accordingly,processor 214 may comprise one or more storage management processors(SMPs). The application program 220 may operate within a single computerand/or DSM 210 or as part of a distributed computer system comprising anetwork of computing devices. The network may encompass one or morecomputers connected via a local area network and/or Internet connection(which may be public or secure, e.g. through a virtual private network(VPN) connection), or via a fibre channel SAN or other known networktypes as will be understood by those skilled in the art.

The present invention may be a system, a method, and/or a computerprogram product. The computer program product may include a computerreadable storage medium (or media) having computer readable programinstructions thereon for causing a processor to carry out aspects of thepresent invention.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, or either source code or object code written in anycombination of one or more programming languages, including an objectoriented programming language such as Smalltalk, C++ or the like, andconventional procedural programming languages, such as the “C”programming language or similar programming languages. The computerreadable program instructions may execute entirely on the user'scomputer, partly on the user's computer, as a stand-alone softwarepackage, partly on the user's computer and partly on a remote computeror entirely on the remote computer or server. In the latter scenario,the remote computer may be connected to the user's computer through anytype of network, including a local area network (LAN) or a wide areanetwork (WAN), or the connection may be made to an external computer(for example, through the Internet using an Internet Service Provider).In some embodiments, electronic circuitry including, for example,programmable logic circuitry, field-programmable gate arrays (FPGA), orprogrammable logic arrays (PLA) may execute the computer readableprogram instructions by utilizing state information of the computerreadable program instructions to personalize the electronic circuitry,in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the block may occur out of theorder noted in the figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

While one or more embodiments of the present invention have beenillustrated in detail, the skilled artisan will appreciate thatmodifications and adaptations to those embodiments may be made withoutdeparting from the scope of the present invention as set forth in thefollowing claims.

What is claimed is:
 1. A method for restoring distributed shared memory(DSM) data consistency within a recovery process from a failure of anode in a cluster of nodes by a processor device, comprising: organizinga DSM component as a matrix of pages, wherein a data structure of a setof data structures occupies a column in the matrix of pages; maintaininga recovery file in a persistent storage, wherein the recovery fileconsists of a plurality of entries and each one of the plurality ofentries corresponds to a column in the matrix of pages by a location ofeach one of the plurality of entries; performing each of: storing theset of data structures in the DSM component and in the persistentstorage; incorporating into each one of the plurality of entries in therecovery file an indication if an associated column in the matrix ofpages is assigned with the data structure of the set of data structures;incorporating into each one of the plurality of entries in the recoveryfile identifying key properties of the data structure of the set of datastructures and a specification of the location of the data structure inthe persistent storage if the associated column in the matrix of pagesis assigned; and defining each column in the matrix of pages to consistof a plurality of pages, wherein at least one page of the plurality ofpages includes the identifying key properties of at least one of aplurality of data structures and additional pages of the plurality ofpages include updatable properties of the at least one of the pluralityof data structures.
 2. The method of claim 1, further includingperforming one of: updating the recovery file during an addition of anew data structure to the set of data structures by performing one of:updating one of the plurality of entries in the recovery file, whichcorresponds by the location to the column in the matrix of pages of therecovery file where the new data structure has been inserted, with theidentifying key properties of the new data structure and with thelocation of the new data structure in the persistent storage, andmodifying those of the plurality of entries in the recovery file, whichcorrespond by the location of the plurality of entries in the recoverythe to the columns in the matrix of pages, wherein the assignments ofthe columns in the matrix of pages with existing data structures havebeen modified during the addition of the new data structure, withupdated identifying key properties and storage locations reflecting thenew assignments.
 3. The method of claim 2, further including modifyingthe recovery file such that the updating is performed in memory and thenwritten to the recovery file in the persistent storage.
 4. The method ofclaim 1, further including using the recovery during the recoveryprocess by performing one of: denoting pages of the plurality of pagesin the matrix of pages, whose contents is invalid, as invalid DSM pages,for an invalid DSM page of the plurality of pages containing identifyingkey properties, reassigning one of the plurality of pages if one of: oneof the entries in the plurality of entries in the recovery filecorresponding to the column in the matrix of pages containing one of theplurality of invalid DSM pages indicates that there is no assignment ofthe column to the data structure in the persistent storage, and the oneof the entries in the plurality of entries indicates that there is theassignment but the data structure does not exist in the specifiedlocation in the persistent storage, then that DSM page is reset to beunassigned, loading contents of the data structure from the persistentstorage into the one of the plurality of pages and setting the one ofthe plurality of pages as assigned to the data structure if the one ofthe entries in the plurality of entries in the recovery filecorresponding to the column in the matrix of pages indicates that thereis the assignment of the column to the data structure in the persistentstorage and there is an actual data structure in the specified locationin the persistent storage, and setting the one of the plurality of pagesas valid.
 5. The method of claim 4, further including, for an additionalinvalid DSM page containing updateable properties, performing one of:resetting the additional invalid DSM page to be unassigned if a DSM pagecontaining the identifying key properties, which is located within asame column in the matrix of pages, is not assigned with the datastructure, loading contents of the data structure from the persistentstorage into the additional invalid DSM page, and setting the additionalinvalid DSM page as assigned to the data structure, if the DSM pagecontaining the identifying key properties, which is located within thesame column in the matrix of pages, is assigned with the data structure,and setting the additional invalid DSM page as valid.
 6. The method ofclaim 4, further including determining a DSM page of the plurality ofpages as invalid if the DSM page is invalid in a local DSM of a nodeperforming a recovery operation and if the DSM page is determined to beinvalid in a plurality of DSMs of all other operational cluster nodes inreply to a query sent by the node to each of the operational clusternodes.
 7. The method of claim 6, further including blocking access,except for the recovery operation, by a DSM mechanism to the pluralityof pages determined to be invalid for a duration of the recoveryoperation.
 8. A system for restoring distributed shared memory (DSM)data consistency within a recovery process from a failure of a node in aduster of nodes in a computing environment, comprising: the DSM; a DSMcomponent in the DSM having set of data structures; a plurality of nodesin the duster of nodes in the DSM; a storage system associating with theduster of nodes, wherein the storage system includes at least persistentstorage; and at least one processor device, operable in the in the DSM,in communication with the plurality of nodes and the storage system,wherein the at least one processor device: organizes a DSM component asa matrix of pages, wherein a data structure of a set of data structuresoccupies a column in the matrix of pages, and maintains a recovery filein a persistent storage, wherein the recovery file consists of aplurality of entries and each one of the plurality of entriescorresponds to a column in the matrix of pages by a location of each oneof the plurality of entries; wherein the at least one processor deviceperforms each of: storing the set of data structures in the DSMcomponent and in the persistent storage; incorporating into each one ofthe plurality of entries in the recovery file an indication if anassociated column in the matrix of pages is assigned with the datastructure of the set of data structures; and incorporating into each oneof the plurality of entries in the recovery file identifying keyproperties of the data structure of the set of data structures and aspecification of the location of the data structure in the persistentstorage if the associated column in the matrix of pages is assigned; anddefines each column in the matrix of pages to consist of a plurality ofpages; wherein at least one page of the plurality of pages includes theidentifying key properties of at least one of a plurality of datastructures and additional pages of the plurality of pages includeupdatable properties of the at least one of the plurality of datastructures.
 9. The system of claim 8, wherein the at least one processordevice performs one of: updating the recovery file during an addition ofa new data structure to the set of data structures by performing one of:updating one of the plurality of entries in the recovery file, whichcorresponds by the location to the column in the matrix of pages of therecovery file where the new data structure has been inserted, with theidentifying key properties of the new data structure and with thelocation of the new data structure in the persistent storage, andmodifying those of the plurality of entries in the recovery file, whichcorrespond by the location of the plurality of entries in the recoverythe to the columns in the matrix of pages, wherein the assignments ofthe columns in the matrix of pages with existing data structures havebeen modified during the addition of the new data structure, withupdated identifying key properties and storage locations reflecting thenew assignments.
 10. The system of claim 9, wherein the at least oneprocessor device modifies the recovery file such that the updating isperformed in memory and then written to the recovery file in thepersistent storage.
 11. The system of claim 8, wherein the at least oneprocessor device uses the recovery the during the recovery process byperforming one of: denoting pages of the plurality of pages in thematrix of pages, whose contents is invalid, as invalid DSM pages, for aninvalid DSM page of the plurality of pages containing identifying keyproperties, reassigning one of the plurality of pages if one of: one ofthe entries in the plurality of entries in the recovery filecorresponding to the column in the matrix of pages containing one of theplurality of invalid DSM pages indicates that there is no assignment ofthe column to the data structure in the persistent storage, and the oneof the entries in the plurality of entries indicates that there is theassignment but the data structure does not exist in the specifiedlocation in the persistent storage, then that DSM page is reset to beunassigned, loading contents of the data structure from the persistentstorage into the one of the plurality of pages and setting the one ofthe plurality of pages as assigned to the data structure if the one ofthe entries in the plurality of entries in the recovery filecorresponding to the column in the matrix of pages indicates that thereis the assignment of the column to the data structure in the persistentstorage and there is an actual data structure in the specified locationin the persistent storage, and setting the one of the plurality of pagesas valid.
 12. The system of claim 11, wherein the at least one processordevice, for an additional invalid DSM page containing updateableproperties, performs one of: resetting the additional invalid DSM pageto be unassigned if a DSM page containing the identifying keyproperties, which is located within a same column in the matrix ofpages, is not assigned with the data structure, loading contents of thedata structure from the persistent storage into the additional invalidDSM page, and setting the additional invalid DSM page as assigned to thedata structure, if the DSM page containing the identifying keyproperties, which is boated within the same column in the matrix ofpages, is assigned with the data structure, and setting the additionalinvalid DSM page as valid.
 13. The system of claim 11, wherein the atleast one processor device determines a DSM page of the plurality ofpages as invalid if the DSM page is invalid in a local DSM of a nodeperforming a recovery operation and if the DSM page is determined to beinvalid in a plurality of DSMs of all other operational cluster nodes inreply to a query sent by the node to each of the operational clusternodes.
 14. The system of claim 13, further including a DSM mechanismcontrolled by the at least one processor device, wherein the DSMmechanism blocks access, except for the recovery operation, by the DSMcomponent to the plurality of pages determined to be invalid for aduration of the recovery operation.
 15. A computer program product forrestoring distributed shared memory (DSM) data consistency within arecovery process from a failure of a node in a duster of nodes by aprocessor device, the computer program product comprising acomputer-readable storage medium having computer-readable program codeportions stored therein, the computer-readable program code portionscomprising: a first executable portion that organizes a DSM component asa matrix of pages, wherein a data structure of a set of data structuresoccupies a column in the matrix of pages; and a second executableportion that maintains a recovery file in a persistent storage, whereinthe recovery file consists of a plurality of entries and each one of theplurality of entries corresponds to a column in the matrix of pages by alocation of each one of the plurality of entries; a third executableportion that performs each of: storing the set of data structures in theDSM component and in the persistent storage; incorporating into eachentry in the recovery file an indication if an associated column in thematrix of pages is assigned with the data structure of the set of datastructures, and incorporating into each entry in the recovery fileidentifying key properties of the data structure of the set of datastructures and a specification of the location of the data structure inthe persistent storage if the associated column in the matrix of pagesis assigned; and a fourth executable portion that defines each column inthe matrix of pages to consist of a plurality of pages, wherein at leaston page of the plurality of pages includes the identifying keyproperties of at least one of a plurality of data structures andadditional pages of the plurality of pages include updatable propertiesof the at least one of the plurality of data structures.
 16. Thecomputer program product of claim 15, further including a fourthexecutable portion that performs one of: updating the recovery fileduring an addition of a new data structure to the set of data structuresby performing one of: updating the entry in the recovery file, whichcorresponds by the location to the column in the matrix of pages of therecovery the where the new data structure has been inserted, with theidentifying key properties of the new data structure and with thelocation of the new data structure in the persistent storage, andmodifying those of the plurality of entries in the recovery file, whichcorrespond by the location of the plurality of entries in the recoveryfile to the columns in the matrix of pages, wherein the assignments ofthe columns in the matrix of pages with existing data structures havebeen modified during the addition of the new data structure, withupdated identifying key properties and storage locations reflecting thenew assignments.
 17. The computer program product of claim 16, furtherincluding a fifth executable portion that modifies the recovery filesuch that the updating is performed in memory and then written to therecovery file in the persistent storage.
 18. The computer programproduct of claim 15, further including a fifth executable portion thatuses the recovery file during the recovery process by performing one of;denoting pages of the plurality of pages with invalid contents that arelocated in the matrix of pages as invalid DSM pages, for an invalid DSMpage of the plurality of pages containing identifying key properties,reassigning one of the plurality of pages if one of: one of the entriesin the plurality of entries in the recovery file corresponding to thecolumn in the matrix of pages indicating that there is no assignment ofthe column to the data structure in the persistent storage, and the oneof the entries in the plurality of entries indicates that there is theassignment but there the data structure does not exist in the specifiedlocation in the persistent storage, then that DSM page is reset to beunassigned, loading contents of the data structure from the persistentstorage into the one of the plurality of pages and setting the one ofthe plurality of pages as assigned to the data structure if the one ofthe entries in the plurality of entries in the recovery filecorresponding to the column in the matrix of pages indicates that thereis the assignment of the column to the data structure in the persistentstorage and there is an actual data structure in the specified locationin the persistent storage, and setting the one of the plurality of pagesas valid.
 19. The computer program product of claim 18, furtherincluding a sixth executable portion that, for an additional invalid DSMpage containing updateable properties, performs one of: resetting theadditional invalid DSM page to be unassigned if a DSM page containingthe identifying key properties, which is located within a same column inthe matrix of pages, is not assigned with the data structure, loadingcontents of the data structure from the persistent storage into theadditional invalid DSM page, and setting the additional invalid DSM pageas assigned to the data structure, if the DSM page containing theidentifying key properties, which is located within the same column inthe matrix of pages, is assigned with the data structure, and settingthe additional invalid DSM page as valid.
 20. The computer programproduct of claim 18, further including a sixth executable portion thatdetermines a DSM page of the plurality of pages as invalid if the DSMpage is invalid in a local DSM of a node performing a recovery operationand if the DSM page is determined to be invalid in a plurality of DSMsof all other operational cluster nodes in reply to a query sent by thenode to each of the operational cluster nodes.
 21. The computer programproduct of claim 20, further including an seventh executable portionthat blocks access by a DSM mechanism, except for the recoveryoperation, by the DSM component to the plurality of pages determined tobe invalid for a duration of the recovery operation.